Ask a few questions when a Magic Compose description doesn't say what people sign up for - #577

Tophat and evals: the prompt that asks

The app. I ran b8f2e66 locally with pnpm dev, Postgres 16, Chromium, and Gemini Flash through OpenRouter. The full flow works, and so do the retry, asked_again, the 12th and 13th draft of the hour, and every activity row. #576 has the screen details.

The server sends the merged system prompt, with both QUESTIONS and CAPACITY. The follow-up message matches #573's format word for word. The server log holds no description or answer text.

Borderline prompts on Flash (made up, 3 tries each):

• Drafts: "Field trip chaperones, 4 needed" and "Carpool for swim meets". The starter examples still draft too.
• Asks: "Volunteers needed" and "Teacher appreciation week". "Help at the school fun run on May 2" asks without "when". "Vente de gâteaux pour l'école" asks with French options.
• Either: "Book fair volunteers" asked once and drafted TBD rows twice.

The evals you asked for. Flash ran 3× per side. A panel of 9 OpenRouter models also ran 3× per side: llama-3.3-70b, llama-3.1-8b, mistral-small-3.2, qwen3-30b-a3b, gemma-3-27b, gpt-oss-120b, deepseek-v4-flash, gpt-4o-mini and claude-haiku-5.5. The comparison is b8f2e66 against main at 4816dc4, plus eval:mcp once per side on Flash. P-values are Fisher's exact test.

The three cases you asked to watch pass on Flash, 4 of 4 each:

• ev_037: asks, then drafts TBD rows and never asks again.
• ev_030: leaves "when" out.
• ev_031: gives French options.

Good

• Asks where it should. 158 of 189 panel replies on the thin prompts (84%) were valid questions. None offered a date or a number, and every one asked all three topics where all three were missing.
• Never asks again after answers. There was no asked_again in 48 real second calls. ev_036's counts were right 27 of 27 times.
• Placeholders improved. On ev_034, 19 of 20 drafts were all TBD rows, against 15 of 26 on main, and none invented dates (main did 3 times).
• main bears out the issue. On the thin prompts, 67% of its drafts are all TBD rows and 33% invent items.

To fix

1. It asks about prompts that already list what people sign up for. This is the main regression.
   • On ev_001–027, 27 of 621 panel replies are questions, against 0 of 621 on main (p=1e-8). On the near misses ev_032–035 it's 24 of 108, against 0 (p=3e-8).
   • The worst were qwen3-30b (12 of 69), plus gpt-4o-mini and mistral (6 of 12 on the near misses). Flash asked on none of the near misses.
   • The model hands the prompt's own items back as options. Here is qwen on ev_001, the supplies list: {"questions":{"topics":["what","how_many"],"whatOptions":["Tissues","Hand sanitizer","Glue sticks","Washable markers"]}}.
   • A fix: rule 4 could say that a list of items, roles or tasks is what people sign up for, and a worked example could show an item list with no date that drafts.
2. A questions reply with the wrong shape becomes an error. 6 of 245 questions replies failed QuestionsSchema: deepseek and mistral asked only ["when","how_many"] when "what" was already known, and gpt-oss offered 5 options. The organizer gets the old "AI draft did not match expected shape — try again or adjust your prompt". The call is charged and nothing retries; I reproduced that through the route with a canned one-option reply. A retry like the follow-up's, or keeping only the first 4 options, would cover most of it.
3. ev_008 is Example C word for word, so it measures recall: 23 of 31 asks copy Example C's options exactly. This PR fixed the same overlap for Example D. Change ev_008's prompt, or Example C's.
4. Eval harness bugs:
   • slot_dates.* reads "TBD: date 1" as a date, so ev_037 fails when it shouldn't (gpt-4o-mini 3 of 3, gemma 1 of 3).
   • The MCP "?" heuristic failed a real request that had no "?" (Flash, ev_034).
   • ev_031's French check passes on articles alone, so off-topic French gets through.
   • ev_002's matcher still rejects a quantity field.

Your call

• Flash now asks on ev_004 ("U12 baseball practice every Tuesday and Thursday …"). It asks 6 of 6 times, plus 3 of 3 in the tophat. main drafts the right 14 slots 6 of 6 times (p=0.002). The options are sensible ("Player attendance", "Assistant coaching", "Snack duty"), and the prompt doesn't say what people sign up for. So either the behaviour or ev_004's expectation should change.
• null capacity shows up where the prompt never said "no limit". That's 84 of 518 panel drafts (16%), against 0 of 538 on main (p=3e-28). Flash does it 7 of 60 times, and so does every other model. It comes from the new schema line "capacity": <integer, 1 by default, or null for no limit>. For example, Haiku on ev_002 (80 cookies) gives quantity: 80 with capacity: null. That's arguably right for item lists, and arguably against "defaults to 1".
• Drafts got slightly worse on ev_001–027. Measured on what each case is about, they pass 83% of the time against 88% on main (p=0.017). The drop is mostly the two llamas. Haiku, deepseek, gpt-oss and gpt-4o-mini are flat.

Weak spots, for the record:

• llama-3.1-8b asks only 4 of 21 times.
• ev_030 asks 14 of 27 times; the other drafts invent items or times.
• 8 of 27 ev_036 drafts split 9 to 1 into hourly shifts (gemma and the llamas).
• gpt-4o-mini turns the given time into "TBD" 3 of 3 times.

Related, older than this stack: one-day drafts keep the date in the description and get no Date column, so their reminders never go out. With the plain description it happened 6 of 6 times on Flash. After this stack's "One day" answer it happened 3 of 8 times. Filed as #685, to follow this PR.

The evals made 2,239 calls for $3.40.

Generated by Claude Code

Thanks for the eval run. It found what my fake endpoint couldn't. Everything below is pushed. I can't run a real model here, so the new wording needs another eval run before merging.

To fix

1. It asked about prompts that already list what people sign up for. Changes are in 1793483 and cc9fc83, and 82c6a92 changes the no-limit wording (3rd bullet).
   • Rule 4 (drafter only; the MCP side is unchanged) now has a sentence for this, and a case that should still ask:

     Named items, roles or tasks, or dated practices, sessions or shifts, already say what: draft, don't ask. An event with no item or role ("family movie night on March 3") doesn't: ask.

   • The options line says: "If the options would be the prompt's own items or roles, it already says what: draft instead." An advisor read flagged that a looser version ("repeat something the prompt lists") could make "Bake sale" draft, so I narrowed it.
   • The QUESTIONS heading now starts "only when the prompt doesn't say what people sign up for (rule 4)".
   • A new Example E shows an item list with no date that drafts (gear to lend for a camping trip). No eval prompt mentions any of its words.
2. A questions reply in the wrong shape became an error. This is fixed in Let Magic Compose answer with questions, and draft from the answers #573 (2e85bcf, b09eb57).
   • Five options, or a repeat, keep the first four different ones. A blank or over-long option, and a topic the page doesn't know, are dropped.
   • Anything else (no "what", or under two usable options) gets one retry on the same quota and time budget, asking for a draft or a refusal.
   • An organizer who left is recorded as aborted.
3. ev_008 copied Example C word for word. It's now a swim club prompt.
   • A new test fails when a worked example, or a rule's "e.g." text, matches an eval prompt, whole words only.
   • That test caught ev_034 too: it contained rule 4's "snack duty for the soccer team". It now says the basketball team.
4. Harness bugs:
   • slot_dates.* now counts only real days, using the converter's own isRealDate. "TBD: date 1", a weekday or "TBD (2026-06-06)" no longer reads as a date.
   • The MCP check now passes a request with no "?" ("Please share the dates…", "I'll need…"). A refusal that closes with "Let me know what else…" still fails.
   • ev_031 needs a French bake-sale word. Articles, generic verbs and words English shares ("stand", "table") don't count, so English options fail.
   • ev_002 accepts a quantity or amount field.

Your call

• ev_004 keeps drafting. A schedule of dated practices is the slots, so I changed the rule rather than the case.
• Null capacity.
  • The schema line is now <integer, 1 by default; null only if the prompt or answers say there's no limit>.
  • Rule 9 now ends: No limit ("unlimited", "as many as want") means null; never null as a guess.
  • Item lists with a count (ev_002's 80 cookies) still get the count. The answers' "no limit" still maps to null.
• Drafts on ev_001–027 slipped (83% vs 88%). I haven't changed anything for this on its own. Part of it is likely the over-asking above, so the next run should show whether it's still there.

Worth watching in the next run

• ev_030 ("Team party on Saturday, June 5") should still ask, because of the new counter-case.
• ev_004, ev_001–027 and ev_032–035 for over-asking.
• How often capacity is null.
• ev_008 and ev_034 with their new prompts.

#685 is noted as the follow-up.

Generated by Claude Code